Systems and methods for modeling the breast using spherical harmonics

ABSTRACT

A system and a computer implemented method are disclosed to model breast shape using three-dimensional spherical harmonics with adjustable parameters to modulate breast size, projection, and/or ptosis. The method includes receiving a 3D image including a breast, identifying the breast in the 3D image, extracting 3D image data of the breast from the 3D image, forming a closed object using the 3D image data of the breast to create a zero-genus surface, mapping the 3D image data of the breast to a predefined template using spherical coordinates, and determining a 3D spherical harmonic descriptor of the 3D image data of the breast.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Stage Application filed under 35U.S.C. § 371(a) claiming the benefit of and priority to InternationalPatent Application No. PCT/US2020/029783, filed on Apr. 24, 2020, whichclaims the benefit of and priority to U.S. Provisional PatentApplication Ser. No. 62/838,997, filed on Apr. 26, 2019, the entirecontents of both applications are incorporated by reference herein.

TECHNICAL FIELD

The present application relates to systems and methods for the modelingof breasts, and in particular, to the modeling of breasts usingspherical harmonics.

SUMMARY

This disclosure relates to systems and methods for the modeling ofbreasts using spherical harmonics. In accordance with aspects of thepresent disclosure, a computer implemented method of modeling breastshape is presented. The method includes receiving a three-dimensional(3D) image including a breast, identifying the breast in the 3D image,extracting 3D image data of the breast from the 3D image, forming aclosed object using the 3D image data of the breast to create azero-genus surface, mapping the 3D image data of the breast to apredefined template using spherical coordinates, and determining a 3Dspherical harmonic descriptor of the 3D image data of the breast basedon a least squares estimation.

In an aspect of the present disclosure, the method further includesidentifying parameters of the 3D spherical harmonic descriptor thatrepresent anatomical breast parameters including at least one of aheight, a width, a depth, or ptosis.

In another aspect of the present disclosure, the method further includesidentifying different types of breast shapes, including at least one ofa natural breast shape, a surgically altered breast shape, an autologousbreast, an implant reconstructed breast, and/or a combination ofautologous and implant breasts, based on spherical harmonic (SPHARM)coefficients.

In an aspect of the present disclosure, the 3D image is a patient'spreoperative image. The method further includes, predicting apost-operative breast shape from the 3D image based on the 3D SPHARMmodel, and outputting a predicted 3D image based on the predictedpost-operative breast shape.

In yet another aspect of the present disclosure, the predicting mayinclude searching a database for a 3D image of at least one secondpatient with similar demographics and/or medical history, to the patientof the received 3D image. The database may include pre-operative andpost-operative 3D images. The predicting may further include locating apre-operative 3D image of a second patient with a similar age, BMI (BodyMass Index), breast size, and/or breast shape, locating a post-operative3D image of the second patient with the similar age, breast size, and/orbreast shape, generating an average pre-operative 3D image based on thepre-operative 3D images, generating an average post-operative 3D imagebased on the post-operative 3D images, determining SPHARM coefficientsof at least one of the average post-operative and/or pre-operative 3Dimage or a located post-operative 3D image, determining SPHARMcoefficients of the received 3D image, determining a difference betweenSPHARM coefficients of the received 3D image and SPHARM coefficients ofthe average post-operative 3D image, and/or determining a differencebetween SPHARM coefficients of the average pre-operative image andSPHARM coefficients of the average post-operative 3D image, applying thedifference in SPHARM coefficients to the received 3D image, and morphingthe breast of the received 3D image based on the determined SPHARMcoefficients.

In yet another aspect of the present disclosure, the predicting mayinclude identifying, in a database, a post-op 3D image of at least onesecond patient with similar demographics or medical history, to thepatient of the received 3D image, wherein the database may includepost-operative 3D images of breasts, generating a templatepost-operative 3D image based on the identified post-operative images torepresent a particular outcome and patient type based on at least one ofage, BMI, ethnicity/race, determining SPHARM coefficients of thereceived 3D image and the SPHARM coefficients of the template,determining a difference between the SPHARM coefficients of the received3D image and the SPHARM coefficients of the template, applying thedifference in SPHARM coefficients to the received 3D image, and morphingthe breast of the received 3D image based on the determined SPHARMcoefficients.

In a further aspect of the present disclosure, the predicting mayinclude using a machine learning algorithm, where training data inputsinclude pre-operation image and/or model data, post operation imageand/or model data, and/or patient demographic data.

In a further aspect of the present disclosure, the machine learningalgorithm may include a neural network, random forest regression, linearregression (LR), ridge regression (RR), least-angle regression (LARS),and/or least absolute shrinkage and selection operator regression(LASSO).

In a further aspect of the present disclosure, the method may includeidentifying different types of breast shapes based on position includingat least one of upright, supine, prone, or any position there between,generating position specific templates. The outputting may be based onpatient position including at least one of upright, supine, prone, orany position there between.

In an aspect of the present disclosure, the different types of breastshapes may include natural, unnatural, surgically altered, and/or aged.

In an aspect of the present disclosure, the forming of a closed objectmay include identifying holes in a first mesh by finding boundary edges,which are edges that are not shared by two faces, calculating the anglebetween adjacent boundary edges at a vertex, and locating the smallestangle and creating a new triangle at the vertex. Creating a second meshto substantially fill the identified holes. A location of a secondvertex may be determined by an average edge length and the shortestdirection to close a gap across the two meshes. Forming of a closedobject may further include computing a distance between every newlycreated vertex and every related boundary vertex, and in a case wherethe distance between them is less than a predetermined threshold theyare merged. Forming of a closed object may further include updating themesh based on the computed distance.

In an aspect of the present disclosure, the 3D image may be a patient'spreoperative image. The instructions, when executed, may further causethe system to: predict a post-operative breast shape from the 3D imagebased on the 3D spherical harmonic (SPHARM) model and output a predicted3D image based on the predicted post-operative breast shape.

In an aspect of the present disclosure, a system for modeling a breastshape includes a processor and a memory. The memory includesinstructions, which when executed by the processor, cause the system toreceive a 3D image including a breast, identify the breast in the 3Dimage, extract 3D image data of the breast from the 3D image, form aclosed object using the 3D image data of the breast to create azero-genus surface, map the 3D image data of the breast to a predefinedtemplate using spherical coordinates, and determine a 3D sphericalharmonic descriptor of the 3D image data of the breast.

In an aspect of the present disclosure, the instructions, when executed,may further cause the system to identify parameters of the 3D sphericalharmonic descriptor that represent anatomical breast parametersincluding a height, a width, a depth, and/or ptosis.

In an aspect of the present disclosure, the instructions, when executed,may further cause the system to identify different types of breastshapes, including natural breast shape, cosmetically altered breastshape, surgically reconstructed breast shape, reduction mammoplasty,reduction mastopexy, augmentation mammoplasty, augmentation mastopexy,or correction of any breast shape deformities, based on sphericalharmonic coefficients.

In an aspect of the present disclosure, the 3D image may be a patient'spreoperative image. The instructions, when executed, may further causethe system to: predict a post-operative breast shape from the 3D imagebased on the 3D SPHARM model and output a predicted 3D image based onthe predicted post-operative breast shape.

In an aspect of the present disclosure, when predicting, theinstructions, when executed, may further cause the system to search adatabase for a 3D image of at least one second patient with similardemographics or medical history, to the received the patient of the 3Dimage, wherein the database includes pre-operative and post-operative 3Dimages, determine SPHARM coefficients of the received 3D image, locate apre-operative 3D image of a second patient with a similar age, breastsize, and/or breast shape based on the SPHARM coefficients, locate apost-operative 3D image of the second patient, generate an averagepre-operative 3D image based on the pre-operative 3D images, generate anaverage post-operative 3D image based on the post-operative 3D images,determine SPHARM coefficients of at least one of the averagepre-operative 3D image, determine SPHARM coefficients of the averagepost-operative 3D image and/or a located post-operative 3D image,determine a difference between SPHARM coefficients of the received 3Dimage and/or the average pre-operative image and SPHARM coefficients ofthe average post-operative 3D image, apply the difference in SPHARMcoefficients to the received 3D image, and morph the breast of thereceived 3D image based on the determined SPHARM coefficients.

In an aspect of the present disclosure, when predicting, theinstructions, when executed, may further cause the system to identify,in a database, a post-op 3D image of at least one second patient withsimilar demographics or medical history, or breast shape to the patientof the received 3D image. The database may include post-operative 3Dimages of breasts. When predicting, the instructions may further causethe system to generate a template post-operative 3D image based on theidentified post-operative images to represent a particular outcome,determine SPHARM coefficients of the received 3D image and the SPHARMcoefficients of the template, determine a difference between the SPHARMcoefficients of the received 3D image and the SPHARM coefficients of thetemplate, apply the difference in SPHARM coefficients to the received 3Dimage, and morph the breast of the received 3D image based on thedetermined SPHARM coefficients.

In an aspect of the present disclosure, the predicting may include usinga machine learning algorithm, where training data inputs include atleast one of pre and post operation image data or patient demographicdata, wherein the machine learning algorithm includes a neural network,random forest regression, linear regression (LR), ridge regression (RR),least-angle regression (LARS), and/or least absolute shrinkage andselection operator regression (LASSO).

In an aspect of the present disclosure, a non-transitory storage mediumthat stores a program causing a computer to execute a method formodeling a breast shape. The method includes receiving a 3D imageincluding a breast, identifying the breast in the 3D image, extracting3D image data of the breast from the 3D image, forming a closed objectusing the 3D image data of the breast to create a zero-genus surface,mapping the 3D image data of the breast to a predefined template usingspherical coordinates, and determining a 3D spherical harmonicdescriptor of the 3D image data of the breast.

Further details and aspects of exemplary embodiments of the presentdisclosure are described in more detail below with reference to theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the features and advantages of the disclosedtechnology will be obtained by reference to the following detaileddescription that sets forth illustrative embodiments, in which theprinciples of the technology are utilized, and the accompanying drawingsof which:

FIGS. 1A-B are diagrams of autologous reconstruction and implantreconstruction, in accordance with aspects of the present disclosure;

FIGS. 2A-C are examples of a 3D image of the front torso, in accordancewith aspects of the present disclosure;

FIG. 3A is an image of a medical imaging system, in accordance withaspects of the present disclosure;

FIG. 3B is diagram of an exemplary controller, in accordance withaspects of the present disclosure;

FIG. 4A is an image of linear and contour distances, in accordance withaspects of the present disclosure;

FIG. 4B is a diagram of volume measurements in customized software, inaccordance with aspects of the present disclosure;

FIGS. 5A-G are examples of five global deformations applied to anasymmetric superquadric, in accordance with aspects of the presentdisclosure;

FIG. 6A is a table of the baseline characteristics of 87 patients;

FIG. 6B is a table of the baseline clinical characteristics of thepatients' breasts, in accordance with aspects of the present disclosure;

FIG. 6C is a table of the demographics of 32 patients who underwent TRAMflap and/or implant reconstruction, in accordance with aspects of thepresent disclosure;

FIG. 6D is a table of the procedure for each breast, in accordance withaspects of the present disclosure;

FIG. 6E is a table of the patient demographics at the time of theirpreoperative image, in accordance with aspects of the presentdisclosure;

FIGS. 7A-B are histograms of the number of vertices along the xdirection and the y direction, in accordance with aspects of the presentdisclosure;

FIGS. 7C-E are midline point detection graphs, in accordance withaspects of the present disclosure;

FIG. 7F is a graph of inframammary fold detection, in accordance withaspects of the present disclosure;

FIG. 8 is a diagram defining ellipse formulas, in accordance withaspects of the present disclosure;

FIGS. 9A-C are diagrams of breast cropping, in accordance with aspectsof the present disclosure;

FIGS. 10A-B are graphs of the unclosed breast and back mesh (A) combinedto form a closed genus-zero object (B), in accordance with aspects ofthe present disclosure;

FIGS. 10C-E are diagrams of the rules for creating triangles, inaccordance with aspects of the present disclosure;

FIGS. 11A-B are diagrams of the front and side views of the breast modeland its associated coordinate system, in accordance with aspects of thepresent disclosure;

FIGS. 12A-E are diagrams of a breast modeled with different degreesusing a spherical harmonic MATLAB (SPHARM-MAT) toolbox, in accordancewith aspects of the present disclosure;

FIG. 13 is a diagram for calculating the root mean square error (RMSE),in accordance with aspects of the present disclosure;

FIG. 14 is a diagram of predicting breast outcomes using the average offive models, in accordance with aspects of the present disclosure;

FIG. 15 is a diagram of a half sphere conversion to a SPHARM model, inaccordance with aspects of the present disclosure;

FIGS. 16A-C are graphs of spherical harmonic (SPHARM) models of a small,a medium, and a large breast, in accordance with aspects of the presentdisclosure;

FIG. 17 is a table of synthetic models generated using differentparameters, in accordance with aspects of the present disclosure;

FIG. 18A is a table summarizing the height, width, depth, and number offaces and vertices of the SPHARM breast models, in accordance withaspects of the present disclosure;

FIG. 18B is a table of the root-mean-squared distance (RMSD) between thesynthetic models, in accordance with aspects of the present disclosure;

FIG. 19A is a table of the RMSE between the synthetic models, inaccordance with aspects of the present disclosure;

FIG. 19B is a table of the Hausdorff distance (HD) between the syntheticmodels, in accordance with aspects of the present disclosure;

FIG. 20A is a table of different height, width, and projection parametersettings, in accordance with aspects of the present disclosure;

FIG. 20B is a table showing different ptosis settings, in accordancewith aspects of the present disclosure;

FIGS. 21A-D are bar plots of the coefficient differences for height,width, depth, and ptosis, in accordance with aspects of the presentdisclosure;

FIGS. 22A-D are graphs of the coefficient differences for height, width,depth and ptosis linearly related to their corresponding parameters, inaccordance with aspects of the present disclosure;

FIGS. 23A-B are boxplots of the ptosis coefficients versus the ptosisgrade rating, in accordance with aspects of the present disclosure;

FIGS. 24A-B are tables of the P-values, in accordance with aspects ofthe present disclosure;

FIGS. 25A-D are scatterplots of selected parameter coefficients vs. theparameter, in accordance with aspects of the present disclosure;

FIG. 26A is a table of the statistics of the RMSE between the groundtruth data set and the reconstructed SPHARM models, in accordance withaspects of the present disclosure;

FIG. 26B is a table of the statistics of the Hausdorff distance betweenthe ground truth data set and the reconstructed SPHARM models, inaccordance with aspects of the present disclosure;

FIGS. 27A-C are diagrams of example SPHARM models based on degree 1, 20,and 50 using level 4 icosahedral subdivision, in accordance with aspectsof the present disclosure;

FIG. 28 is a table of the magnitude of the SPHARM coefficients fordifferent degrees, in accordance with aspects of the present disclosure;

FIG. 29 is a table of the P-values from the Wilcoxon rank-sum testcomparing the five coefficient values between different degrees, inaccordance with aspects of the present disclosure;

FIG. 30 is a table of the average percent difference in the SPHARMcoefficients correlated to height, width, depth, and ptosis betweendifferent degrees relative to the SPHARM coefficient, in accordance withaspects of the present disclosure;

FIGS. 31A-C are diagrams of the selection area of the transition pointand lateral point, in accordance with aspects of the present disclosure;

FIG. 32 is a diagram of examples of breast cropping selecting theextreme corners for the transition and lateral points, in accordancewith aspects of the present disclosure;

FIG. 33 is a table of the rate of successful processing at each step ofthe algorithm for different landmark positions, in accordance withaspects of the present disclosure;

FIG. 34 is a table of the Euclidean distance of the user selectionrelative to the ground truth selection for 10 images, in accordance withaspects of the present disclosure;

FIG. 35 is a table of the Euclidean distance of the corner selectionsrelative to the ground truth selection, in accordance with aspects ofthe present disclosure;

FIG. 36 is a diagram of the lateral point location of the different setsrelative to the ground truth, in accordance with aspects of the presentdisclosure;

FIG. 37 is a table of the rate of successful processing at each step ofthe algorithm for different lateral point locations based on the 161breasts, in accordance with aspects of the present disclosure;

FIG. 38 is a diagram of the transition point location of the differentsets relative to the ground truth, in accordance with aspects of thepresent disclosure;

FIG. 39 is a table of the rate of successful processing at each step ofthe algorithm for different transition point locations based on 161breasts, in accordance with aspects of the present disclosure;

FIG. 40 is a table depicting statistics on the projection percentdifference between the ground truth selection and shifting thetransition point (TP) 3 and 6 mm in different directions, in accordancewith aspects of the present disclosure;

FIG. 41 is a table depicting statistics on the volume percent differencebetween the ground truth selection and shifting the transition point(TP) 3 and 6 mm in different directions, in accordance with aspects ofthe present disclosure;

FIG. 42 is a table depicting statistics on the projection percentdifference between the ground truth selection and shifting the lateralpoint (LP) 3 and 6 mm in different directions, in accordance withaspects of the present disclosure;

FIG. 43 is a table depicting statistics on the volume percent differencebetween the ground truth selection and shifting the lateral point (LP) 3and 6 mm in different directions, in accordance with aspects of thepresent disclosure;

FIGS. 44A-B are images of an original breast (FIG. 44A) and modifiedbreast (FIG. 44B) by changing the SPHARM coefficients for height, width,projection, and ptosis, in accordance with aspects of the presentdisclosure;

FIGS. 45A-B are images of an example of a bilateral TRAM flapreconstruction (FIG. 45A) and a bilateral implant reconstruction (FIG.45B), in accordance with aspects of the present disclosure;

FIG. 46A is a set of confusion matrices for k-nearest neighborclassification, quadratic discriminate analysis, and Naïve Bayesclassifier using the SPHARM coefficients, in accordance with aspects ofthe present disclosure;

FIG. 46B is a set of confusion matrices for k-nearest neighborclassification, quadratic discriminate analysis, and Naïve Bayesclassifier using BMI, breast volume, and breast dimensions, inaccordance with aspects of the present disclosure;

FIG. 47 is a scatterplot of the first three principal components forTRAM flap and implant reconstructed breasts, in accordance with aspectsof the present disclosure;

FIG. 48 is a set of images of true versus predicted class reconstructedbreasts according to a TRAM flap or implant, in accordance with aspectsof the present disclosure;

FIGS. 49A-B are depictions of examples of a TRAM reconstructed breast(reference) and its four nearest neighbors based on the RMSD, inaccordance with aspects of the present disclosure;

FIGS. 50A-B are depictions of examples of an implant reconstructedbreast and its four nearest neighbors based on the RMSD, in accordancewith aspects of the present disclosure;

FIG. 51A is a graph of the front and side views of the average breastmodel, in accordance with aspects of the present disclosure;

FIG. 51B is a graph of the front and side views of the average breastmodel, in accordance with aspects of the present disclosure;

FIG. 52A is a table of the procedures conducted on the left and rightbreast of each patient and the average RMSD for each set of patientswith the same procedures, in accordance with aspects of the presentdisclosure;

FIG. 52B is a table of the statistics on the RMSD comparingpreoperative, postoperative, and predicted breast shapes, in accordancewith aspects of the present disclosure;

FIG. 52C is a table of the RMSD between the true postoperative modelsand the predicted models organized into three BMI groups, in accordancewith aspects of the present disclosure;

FIG. 52D is a table of the Hausdorff distance between the truepostoperative models and the predicted models organized into three BMIgroups, in accordance with aspects of the present disclosure;

FIG. 53 is a table depicting examples of predicted postoperative models,in accordance with aspects of the present disclosure;

FIG. 54 is a diagram of the best prediction out of 53 reconstructedbreasts, in accordance with aspects of the present disclosure;

FIG. 55 is a diagram of the worst prediction out of 53 reconstructedbreasts, in accordance with aspects of the present disclosure;

FIG. 56 is a table of RMSD comparisons between the pre-op breast shapeSPHARM coefficients, the transformed pre-op breast coefficient, andactual post—op coefficients for three patients, in accordance withaspects of the present disclosure;

FIG. 57A-D are images of pre-op (P1), post-op (P2), estimate (E) andoverlay of estimate on the post-op breast, in accordance with aspects ofthe present disclosure; and

FIG. 58 is a block diagram of a method for modeling a breast, inaccordance with the present disclosure.

Further details and aspects of various embodiments of the presentdisclosure are described in more detail below with reference to theappended drawings.

DETAILED DESCRIPTION

This disclosure relates to the modeling of breasts using sphericalharmonics.

Although the present disclosure will be described in terms of specificembodiments, it will be readily apparent to those skilled in this artthat various modifications, rearrangements, and substitutions may bemade without departing from the spirit of the present disclosure. Thescope of the present disclosure is defined by the claims appendedhereto.

For purposes of promoting an understanding of the principles of thepresent disclosure, reference will now be made to exemplary embodimentsillustrated in the drawings, and specific language will be used todescribe the same. It will nevertheless be understood that no limitationof the scope of the present disclosure is thereby intended. Anyalterations and further modifications of the inventive featuresillustrated herein, and any additional applications of the principles ofthe present disclosure as illustrated herein, which would occur to oneskilled in the relevant art and having possession of this disclosure,are to be considered within the scope of the present disclosure.

As the number of cosmetic and reconstructive breast surgeries performedhas been steadily increasing over the years, there is a greater need forimproved technologies, such as developing a computationalthree-dimensional breast model. Referring to FIGS. 1A-B, two generaltypes of cosmetic and reconstructive breast procedures are shown. Twocommon procedures performed are autologous reconstruction using atransverse rectus abdominis (TRAM) flap is shown in FIG. 1A and implantreconstruction is shown in 1B. In order to improve the results of thesesurgeries, computational three-dimensional breast model may be used.According the present disclosure, a method using spherical harmonics isemployed to produce the three-dimensional breast models.

Spherical harmonics are a complete series of orthogonal functionsdefined on the surface of a sphere. The spherical harmonic (SPHARM)method, converts a 3D object of spherical topology into three sets ofSPHARM coefficients that describe its overall shape in terms of threesets of spherical harmonics basis functions (one set for eachdimension). SPHARM has applicability to several fields includingcomputer vision and computer graphics but has been most aptly applied instudying medical images, particularly in brain morphometry. There areseveral inherent properties of this shape descriptor that make it anadvantageous method for medical image analysis. It can be compactlyrepresented, allows for efficient shape comparison, incorporatesimplicit interpolation since the spherical domain is continuous, and canbe used to establish surface correspondence. In addition, it can beprocessed in both the spatial and frequency domain.

A 3D imaging system may be used to record 3D images of the frontalportion of female torsos. The imaging system may represent a 3D image inthe form of a triangular mesh that contains, for example about 75,000vertices and 140,000 faces for each patient image. Each face also hastexture associated with it, so a 3D texture image can also be viewed.The vertices may be spaced about 3 mm apart from each other and thestated error is less than 0.5 mm. Referring to FIGS. 2A-C, the 3Dsurface image can be viewed as a 3D point cloud (FIG. 2A), a triangularmesh (FIG. 2B), or a 3D texture image overlaid on the triangular mesh(FIG. 2C).

With reference to FIG. 3A an exemplary imaging system 300 is shown. Theexemplary imaging system 300 may include six modular camera units 302and may use, for example, stereophotogrammetry to estimate a 3D surfaceimage from pairs of 2D photographs. These camera units 302 may beeffectively positioned around a person (top, middle, and bottom) toimprove the 3D surface coverage of the front torso. Each camera unit 302may be equipped with a pair of stereo cameras and a color camera. Thestereo cameras may be synchronized to initiate together and has a 1.5millisecond capture speed. A half millisecond later, the color camerasmay be triggered to capture 2D photographs from six differentviewpoints. The system then employs a sophisticated software algorithmto unify the images from the six camera units. Once the 3D surface imageis fully generated, the color photographs are mapped to the generatedtriangular mesh. The illustrated example uses stereophotogrammetry,however, other forms of 3D imaging (such as a portable handheld 3Dscanner, or IPAD Pro, or others) are contemplated.

Referring now to FIG. 3B, there is shown an illustration of exemplarycomponents in the controller 200 of FIG. 3A, in accordance with aspectsof the present disclosure. The controller 200 includes, for example, adatabase 210, one or more processors 220, at least one memory 230, and anetwork interface 240.

The database 210 can be located in a storage. The term “storage” mayrefer to any device or material from which information may be capable ofbeing accessed, reproduced, and/or held in an electromagnetic or opticalform for access by a computer processor. A storage may be, for example,volatile memory such as RAM, non-volatile memory, which permanently holddigital data until purposely erased, such as flash memory, magneticdevices such as hard disk drives, and optical media such as a CD, DVD,Blu-ray disc, cloud storage, or the like.

A database of 3D torso images of breast reconstruction patients andvolunteers and demographic data (e.g., age, BMI, race, previoussurgeries, diagnosis, etc.) may be used. The database may also include anon-limiting list of, breast size, breast shape, breast feeding,gravidity and parity, and medical history. The patients may consist ofthose who underwent autologous reconstruction (TRAM flap, latissimusdorsi flap (LD), and deep inferior epigastric perforators (DIEP) flap)and implant-based reconstruction (saline or silicone implants). They mayhave also received additional procedures, such as mastopexy (forsymmetry with the reconstructed breast and reducing ptosis), fatgrafting, tissue expander placement, and nipple reconstruction. Tissueexpanders, or temporary saline implants, are sometimes used to slowlystretch the skin and pectoralis muscle (large chest muscle). They may belater replaced with tissue or a permanent implant. Within the database,the average age of the patients was 49.9±10.3 years (range: 24 to 75),and BMI was 28.0±5.5 (range: 18.1 to 69.8). During imaging, patients maybe generally in a standing position with their hands on their hips.Images may be taken, typically but not consistently, at three-monthintervals (for up to a period of two years) during the reconstructiveprocess. Some, but not all, patients may have had a preoperative image.

Referring to FIGS. 4A-B, a Java-based visualization and analysis tool(customized software), may be used to manipulate 3D images of the femaletorso (translate, rotate, scale, and crop), manually mark fiducialpoints, such as the sternal notch, nipples, lateral points, inframammaryfold (IMF) points, midline point, transition points, and umbilicus, andextract linear, contour, and volume measurements. Customized softwaremay be used to align the images in the xyz coordinate space, manuallymark fiducial points, and calculate breast volumes. FIG. 4A is anexample image of customized software showing linear and contourdistances. FIG. 4B is an example image of customized software showingvolume measurements of a front female torso.

As the number of cosmetic and reconstructive breast surgeries performedhas been steadily increasing over the years, there is a greater need forimproved technologies, such as developing a computationalthree-dimensional breast model. A breast model for simulating,evaluating, and interactively adjusting breast shape will be a tool forsurgeons in surgical planning and for clinical consultations withpatients in shared decision making.

Breast models may be used to predict breast deformations subject tovarious gravitational positions. For example, breast surgical proceduresare typically performed on a patient lying down, but the patient mayhave to be moved into an upright position a number of times for thesurgeon to assess the shape of the breast. Breast models may also beused to predict breast deformation due to compression from differentimaging modalities in order to view the breasts in an uncompressed stateor to register images. For example, in diagnostic imaging, mammogramsshow two-dimensional projections of the compressed breast while MRIshows three-dimensional images of the breast in the prone position.Registering the two types of images may assist radiologists inmultimodal diagnosis and help in localizing structures, such as tumors.Other reasons for breast simulation models may include planning andrehearsing surgeries, predicting outcomes, and testing new methods andtechniques.

Conversely, very few parametric breast models have been proposed. Forexample, a parametric breast model proposed by Chen, may be used eitherto create a breast model or fit a model to breast data. They initiallycreate an asymmetric superquadric and then perform five globaldeformations to model five major features of breast shape.

Referring to FIGS. 5A-G, these five global deformations/features arecalled lower pole deformation, upper pole deformation, horizontaldeviation deformation, medial deformation, and axillary taildeformation. FIG. 5A and FIG. 5B show the front and side views of abreast model without any deformations applied. FIG. 5C shows a frontview example of lower pole deformation, which models breast sagging.FIG. 5D is a side view example of upper pole deformation, which controlsthe slope and curvature of the upper half of the breast. In FIG. 5E, afront view example is shown of horizontal deviation deformation, whichcontrols the turn of the breast whether to the right and left. FIG. 5Fis a front view example of medial deformation, which flattens out thesides of the breast. FIG. 5G is a front view example of axillary taildeformation, which adjusts the top half of the breast to point towardthe shoulder. The 17 parameters that control the breast size and shapecan be used to quantitatively analyze the degrees of key shape variablesof the breast. However, Chen's model had limited capability inaccurately fitting to breast data due to its basic design. Breastscontain many different curves and shape features that an asymmetricsuperquadric and five global deformations is unable to fully capture.Still, due to the strict control of breast shape and size through the 17parameters, it is helpful to simulate different elliptical models toevaluate the parameters of the disclosed spherical harmonic model andcorrelate the parameters to clinically relevant parameters, such asbreast height, width, depth, and ptosis. The modeling technique that isintroduced in this disclosure may allow for computing the breast shape“distance” based on coefficients, and the coefficients can be related tospecific breast shapes. The breast shapes may include for example,reduction mammoplasty, reduction mastopexy, augmentation mammoplasty,augmentation mastopexy, and/or correction of any breast shapedeformities, based on spherical harmonic coefficients.

The disclosed method allows for modeling the original breast shape(accuracy of fitted model), can be used to modify breast shape (shapemodulation), establishes correspondence between different breast shapesand sizes (cross model alignment), can be used to compute shape distancebetween different breasts (parametric shape comparison), and can be usedto predict breast shapes.

Surgeons may evaluate breast shape by taking measurements in personusing a tape measure in the clinic or using rulers on standardphotographs of the frontal and lateral views. These measurements includethe linear distance between fiducial points, including the sternalnotch, nipples, lateral points, mid-clavicle points, midline, andinframammary fold. Another measurement to describe breast shape isptosis, which is used to describe sagging of the breasts. Theinframammary fold is often designated as a reference point forevaluating the degree of ptosis. However, grading ptotic breasts can bedifficult as the inframammary fold is hidden when the woman is standingin an upright position. Using a modified Regnault's classification ofptosis, the breasts may be assigned a grade ranging from 0 to 3, whereGrade 0 represents no ptosis and Grade 3 represents extreme ptosis. Oneof skill in the art would know what Regnault's classification is and howto implement it. With the advent of three-dimensional imagingtechnology, additional objective measurements, such as surface contoursand curvature, surface area, volume, and even ptosis can now bequantitatively assessed.

In the illustrated embodiment, the SPHARM modeling method was firsttested on three-dimensional preoperative torso surface images of anumber of women scheduled to undergo mastectomy for the treatment orprevention of breast cancer and other abnormalities. None of them haveprevious breast surgeries but may have had a biopsy that did not affectthe breast appearance as determined by an experienced plastic surgeon.Patients with rare congenital breast abnormalities, previous radiationtherapy, or previous major breast surgeries were excluded.

Referring to FIGS. 6A-E, tables of patient demographics, clinicalcharacteristics of the patients' breasts, patient variables, proceduretypes, and the procedures underwent are shown. FIG. 6A is a table of thepercentages of the patient demographics and characteristics describingthe age, BMI, tumor size, race, ethnicity, diagnosis and number treatedwith pre-operative chemotherapy. FIG. 6B is a table of the baselineclinical characteristics of breasts (N=76) for the 73 cancer patients,describing the percentage of the various tumor types and the positionsof the tumors. FIG. 6C is a table of the demographics of 32 patients whounderwent TRAM flap and/or implant reconstruction. FIG. 6D is a table ofthe procedure used for each breast. FIG. 6E is a table of the patientdemographics at the time of their preoperative image.

In various embodiments, three-dimensional breast images of patients whounderwent unilateral or bilateral TRAM flap and/or implantreconstruction were selected, and SPHARM models were generated from theimages. The generated SPHARM models of the reconstructed breasts wereclassified using standard classification methods: k-nearest neighbor,Naïve Bayes, and quadratic discriminant analysis. In variousembodiments, a dataset consisting of a number of patients, includingtheir preoperative and corresponding postoperative images, was createdfor testing predictive modelling. For the preoperative image set and thepostoperative image set, a number of SPHARM models were generated foreach set.

In various embodiments, the breasts may be extracted from the torsoimages by identifying the borders of each breast. In variousembodiments, the fiducial locations at the top, bottom, left, and rightsides of each breast that would delineate how the breast would besegmented from the torso images may be identified. Before identifyingthese fiducial locations, the images had to be manually aligned so thatthe height of a patient from head to foot aligned with the y axis, thewidth from the right shoulder to the left shoulder aligned with the xaxis, and the body faced the positive z direction. Then four fiduciallocations were found along the border of each breast from the 3D torsoimage (FIG. 9A). Two locations were automatically detected: the midlinepoint and inframammary fold. The other two locations, the transitionpoint and the lateral point, may be manually selected. These points maybe manually selected as the breast is usually relatively smooth in theseregions.

Referring to FIGS. 7A-E, the graphical models of the breasts may be madeby finding the number of vertices along the x and y axes (7A-B) and themidline point (7C-E) to show the inframammary folds (7F). Manualfiducial point selection may be conducted in customized software (acustomized Java-based visualization tool), and the automaticallydetected points may be found using code developed in MATLAB 2015a. Toautomatically locate the midline point, the head, arms, and the torsobelow the breasts may be cropped out first. The head and arms containedless points than the torso, and the number of points peaked on the sidesof the torso. Therefore, using a histogram of the number of points alongthe width of the torso in bins of 20 millimeters, the left and rightborders of the torso may be determined as shown in FIG. 7A. Using thissame concept, another histogram may be generated measuring the number ofvertices along the y direction in bins of 20 millimeters to determinethe top cutoff point as shown in FIG. 7B, where the neck was found tohave fewer points relative to the torso. To set the bottom cutoff point,the surface normal in the vertical y direction (corresponding to height)of the mesh vertices may be used to find the lowest visible point of thebreasts. The vertices with surface normal within 18.2° (acos(0.95)) ofthe negative y direction were sub-selected. Then the vertex with thelowest y value was designated as the lowest visible point of the breastand was used as the bottom cutoff point. If there were no valid verticeswith a y surface normal within 18.2° (acos(0.95)) of the negative ydirection, excluding the nipple, then the bottom cutoff point was set to33 centimeters below the top cutoff point. It may be determined that thebreasts were within 33 centimeters from the top cutoff point. Thismostly occurred for relatively small, non-ptotic breasts. Then Gaussiancurvature was computed on the remaining mesh contained within thedesignated borders as shown in FIG. 7C. The area between the breasts,where the midline point is, has negative curvature (concave), while thebreasts themselves have positive curvature (convex). The points withnegative curvature between the two nipples, or most projecting points onthe left and right halves of the torso, were selected as possiblemidline points. The midline point is not more than 10 mm below the mostprojecting points. Then the selected points were separated into 5 mmbins along the y direction, and the width (x range) of each bin wasfound, as shown in FIG. 7D. The bin with the smallest width was selectedand its midpoint was designated as the midline point, shown as a bluestar in FIG. 7E. Next, using a contour detection algorithm, which alsoutilizes Gaussian curvature, points may be estimated along the IMF(inferior breast-chest contour) of the left and right breasts. The redline in FIG. 7F delineates the estimated IMF. The partial torso image inFIG. 7F is colored by the shape index for each vertex in the mesh, whichwas calculated from the contour detection algorithm. The IMF wasdetected by following the negative curvature path along the underside ofthe breast. However, for breasts with prominent nipples that havesignificant negative curvature along the underside of the nipple, thealgorithm selected the underside of the nipple as the inframammary fold,hence providing incorrect fiducial points. In order to avoid selectingthese incorrect fiducial points, the algorithm was modified to ignorethe curvature values of the nipple area. The nipple area wasexperimentally determined to be within 2.5 centimeters of the mostprojected points along the z axis on the left and right half of thetorso. The nipple diameter for women aged 20-64 years may range from forexample, 1 cm to 2.75 cm.

Referring to FIG. 8, in various embodiments, to define the upper borderof each breast, the ellipse formula (using only x and y coordinatesassuming that the x axis aligns with the width of the breast and the yaxis aligns with the breast height):

${{\frac{\left( {x - x_{0}} \right)^{2}}{a^{2}} + \frac{\left( {y - y_{0}} \right)^{2}}{b^{2}}} = 1},$

was used to locate intermediate points between the lateral point (L) andthe transition point (TP), and for connecting the midline point (M) andthe TP.

FIGS. 9A-C are diagrams of breast cropping, in accordance with aspectsof the present disclosure. Dijkstra's shortest path algorithm was usedto connect all the points, defining the estimated closed border of thebreast as displayed in FIG. 9B. One of skill in the art would befamiliar with Dijkstra's shortest path algorithm and understand how toimplement it. The white line depicts the border formed using the ellipseformula, and the black line depicts the border formed using the contourdetection algorithm. Finally, FIG. 9C shows an example of two breastsurface patches that were extracted from the torso image.

To identify which vertices to extract, an initial vertex was selectedbased on the shortest distance to the average of all the border points.Then neighboring vertices directly connected to the initial vertex andthe next neighboring vertices connected those vertices may beiteratively added until the border vertices were selected in which casethe iteration is stopped.

In various embodiments, SPHARM may require a genus zero surface and arelatively dense mesh to accurately model an object. Since the croppedbreast was an unclosed surface patch, a method to patch the back hole inorder to form a closed surface may be used. First, the cropped breastmesh may be pre-processed to clean up non-manifold vertices (i.e., edgesthat are shared by more than two faces and isolated pieces (disconnectedvertices and edges)). Then the advancing front mesh (AFM) technique maybe used to fill any small holes that were created due to the removal ofthe non-manifold vertices following the rules for creating triangles asshown in FIGS. 10C-E. FIG. 10C shows the rule for the condition θi<90°;FIG. 10D shows the rule for the condition when 90°≤θi<150°; and FIG. 10Eshows the rule for the condition when θi≥150 After that, a single-meshsupplement method was used to smooth the border of the breast mesh byconnecting adjacent boundary edges whose angle was less than 90°. Thenthe breasts may be rotated so that the height of the breast aligned withthe z axis, the width aligned with the y axis, and the depth alignedwith the x axis.

Referring to FIGS. 10A-B, graphs of the unclosed breast mesh and backmesh (A) are combined to form a closed genus-zero object (B). To patchthe hole on the backside of the breast surface mesh, a 2D rectangulary-z grid of points was generated based on the size of the hole, and thepoints were equally spaced according to the average length between theboundary points. Then the points outside of the hole boundary may beremoved. Next, Delaunay triangulation was used to connect the generatedpoints to form a mesh. Each point was assigned an x value based on theaverage x value of the breast border points weighted by the 2D Euclideandistance using the y and z coordinates. Then the new mesh may beattached to the breast mesh, and the remaining gaps were filled using ahole filling algorithm based on the advancing front mesh technique asshown in FIGS. 10A-B. In various embodiments, the method may includefirst identifying holes in the mesh by finding boundary edges, which areedges that are not shared by two faces. Boundary edges that form aclosed loop constitute a hole. Next, calculating the angle θ_(i) (0 to360°) between adjacent boundary edges (e_(i) and e_(i+1)) at each vertexvi. Next, finding the smallest angle θ_(i) and create new triangle(s) atvertex vi following the rules shown in FIGS. 10C-E. The location of thenew vertices is determined by the average edge length and the shortestdirection to close the gap across the two meshes. Next, computing thedistance between every newly created vertex and every related boundaryvertex; if the distance between them is less than the given threshold(such as the average edge length), they are merged. Next, update thefront. Next, repeat until all holes are filled.

In order to obtain spherical topography and a standardized orientationof the modeled breasts, the breast image data were mapped to a specifictemplate using spherical coordinates (θ, ϕ) as displayed in FIGS. 11A-B.To do this, each vertex in the breast mesh was bijectively mapped to theunit sphere (θ=[−π/2, π/2] and ϕ=[−π, π]) based on certain landmarks.The nipple (or highest projecting point) was set to θ=π/2. The breast'sboundary points, at the discontinuity between the breast surface meshand the connected back mesh, were assigned θ=0. To determine ϕ for eachpoint, a straight line (CN) extending from the centroid of the breastboundary points (backside of the breast) to the nipple, or mostprojecting point may be generated. Each point (x, y, z) on the breastsurface was then projected onto the CN line, to generate its position(y1, z1) on the line, such that x1, the x-coordinate of the point on theCN line, is equivalent to the x-coordinate of point PP. Then the angle ϕcan be calculated based on the (y, z) breast point coordinate and point(y1, z1) on the CN line. Following determination of ϕ, the set of allthe points on the breast surface having the same ϕ, on the positive(front) side of the breast, are used to trace its angular path along thesurface, and its length is determined. The relative surface distance ofthe point P with respect to this length is scaled by π/2 and is assignedto θ accordingly. The front side of the breast was assigned positivevalues for θ, and the back side was assigned negative values for θ.Also, since the surface of different objects are aligned throughspherical parameterization, achieving correspondence across twodifferent breast models is feasible, which allows for comparisons oflocal and global changes in breast shape and size.

SPHARM Expansion

The Fourier spherical harmonics Y(θ, ϕ) (or SPHARM functions) of degreel and order m can be defined by

${Y_{l}^{m}\left( {\theta,\varnothing} \right)} = {\sqrt{\frac{\left( {{2l} + 1} \right){\left( {l - m} \right)!}}{4{{\pi\left( {l + m} \right)}!}}}{P_{l}^{m}\left( {\cos\theta} \right)}e^{im\varnothing}}$

Where P_(l) ^(m)(cos θ) are the associated Legendre polynomials definedby the differential equation:

$P_{l}^{m} = {\frac{\left( {- 1} \right)^{m}}{\left( {2^{l}l!} \right)}\left( {1 + x^{2}} \right)^{m/2}\left( \frac{d^{l + m}}{dx^{l + m}} \right)\left( {x^{2} - 1} \right)}$

The SPHARM expansion takes the form: v(θ, ∅)=Σ_(l=0) ^(∞)Σ_(m=−m)^(l)c_(l) ^(m)Y_(l) ^(m)(θ, ∅).

Where v(θ, ∅)=(x(θ, ∅), y(θ, ∅), z(θ, ∅))^(T) and c_(l) ^(m)=(c_(lx)^(m), c_(ly) ^(m), c_(lz) ^(m))^(T) are the estimated SPHARMcoefficients. L_(max) is a user-specified degree. The function v(θ, ∅)can be independently decomposed into three functions for the threecoordinates:

${{x\left( {\theta,\varnothing} \right)} = {\sum\limits_{l = 0}^{L_{\max}}{\sum\limits_{m = {- l}}^{l}{c_{lx}^{m}{Y_{l}^{m}\left( {\theta,\varnothing} \right)}}}}};$${{y\left( {\theta,\varnothing} \right)} = {\sum\limits_{l = 0}^{L_{\max}}{\sum\limits_{m = {- l}}^{l}{c_{ly}^{m}{Y_{l}^{m}\left( {\theta,\varnothing} \right)}}}}};{and}$${{z\left( {\theta,\varnothing} \right)} = {\sum\limits_{l = 0}^{L_{\max}}{\sum\limits_{m = {- l}}^{l}{c_{lz}^{m}{Y_{l}^{m}\left( {\theta,\varnothing} \right)}}}}},$

The function values, x_(i,j)=Y_(l) ^(m)(θ_(i), φ_(i)) for 1≤i≤n, areinputted into a linear system for each of the three coordinate functionsas follows:

${{\begin{pmatrix}y_{1,1} & y_{1,2} & \ldots & y_{1,k} \\y_{2,1} & y_{2,2} & \ldots & y_{2,k} \\ \vdots & \vdots & & \vdots \\y_{n,1} & y_{n,2} & \ldots & y_{n,k}\end{pmatrix}\begin{pmatrix}a_{1} \\a_{2} \\ \vdots \\a_{k}\end{pmatrix}} = \begin{pmatrix}x_{1} \\x_{2} \\ \vdots \\x_{n}\end{pmatrix}},$

where y_(i,j)=Y_(l) ^(m)(θ_(i), φ_(i)), j=l²+l+m+1, and k=(L_(max)+1)².Lmax is the user specified degree, degree l=0, . . . , Lmax, and orderm=−l, . . . ,0, . . . , l. Given the xyz coordinates of an object, thecoefficients (a₁, . . . , a_(k))^(T) can be solved for through leastsquares fitting. Increasing degree Lmax increases the number ofcoefficients and provides a more detailed reconstruction. As such, theSPHARM coefficients make up a hierarchical surface descriptor. Thenumber of coefficients is equal to (L_(max)+1)²×3. These coefficientsapproximate the full underlying surface, which can be used to representand reconstruct the object.

The SPHARM descriptor (or set of coefficients) was computed usingSPHARM-MAT toolbox. The SPHARM-MAT toolbox includes methods to performspherical parameterization (a different method from the parameterizationdiscussed above), expansion, registration, and statistical analysis andother utilities. Besides inputting the breast image data to calculatethe SPHARM descriptor, a degree setting needs to be given. Eachadditional degree increases the number of coefficients and therebyincreases the level of detail as shown in FIGS. 12A-E.

To evaluate the fitting accuracy of the SPHARM model to the originalbreast data, the root mean square error (RMSE) between the points of theoriginal breast data and the points of the SPHARM model via theEuclidean distance may be used. The RMSE between the original breastdata points x₁ and the SPHA-RM model points x₂ is defined as

${RMSE} = \sqrt{\frac{1}{n}{\sum\limits_{l = 0}^{n}{\left( {{x1},{i - {x2}},i} \right)^{\land}2}}}$

FIG. 13 is a diagram for calculating the root mean square error (RMSE)1300, in accordance with aspects of the present disclosure. The detailsof how the RMSE is computed for resampled points on the faces of thesurface mesh, instead of the triangle vertices, as discussed below atstep 1306. Initially at step 1302, given a closed breast object (input),the object is parameterized so that each vertex is assigned a sphericalcoordinate (P_(vertices)(x, y, z)→P_(vertices) (θ, ϕ)). Next at step1304, the SPHARM coefficients (output), c, are estimated by solving alinear system using the spherical harmonic equations, Y_(l) ^(m)(θ, ϕ),and the given vertices, P_(vertices) (θ, ϕ), y(θ, ϕ), z(θ, ϕ). Next, atstep 1306, since the SPHARM coefficients are fitted (or over-fitted) tothe input vertices, new vertices by finding the centroid of each face inthe closed breast object (input) may be used, which gives us a new setof vertices (P_(faces) (x′, y′, z′)) and the corresponding parameterizedspherical coordinates (P_(faces) (θ′, ϕ′)). Next at step 1308, P_(faces)((x′θ, ϕ), y(θ, ϕ), z(θ, ϕ)) is estimated using the SPHARM coefficientsfrom step 2 and the spherical coordinates P_(faces) (θ′, ϕ′),

P _(faces) ^(∧)(x′, y′, z′)=Σ_(l=0) ^(L) ^(Max) Σ_(m=−l) ^(l) c _(l)^(m) −Y _(l) ^(m)(θ, ϕ′).

Next at step 1310, calculate the RMSE between P_(faces) (x′, y′, z′) andP^(∧) _(faces) (x′, y′, z′).

The SPHARM coefficients can be used to determine shape similaritybetween objects, such as between preoperative and postoperative breasts,left and right breasts, and the breasts of different patients using asimilarity measure called root mean square distance (RMSD). The RMSD forthe coefficients is:

${RM{SD}} = \sqrt{\frac{1}{4\pi}{\sum\limits_{l = 0}^{L_{\max}}{\sum\limits_{m = {- l}}^{l}{{c_{1,l}^{m} - c_{2,l}^{m}}}^{2}}}}$

Where c_(1,l) ^(m) and c_(2,l) ^(m) are SPHARM coefficients of thebreast shapes being compared.

For set of patients' pre-op breast images whose post-op shapes areknown, the SPHARM coefficients of each breast are computed. Letx=[x₁,x₂, . . . , x₁₃₂₀]^(T) be the spherical coefficients obtained frommodeling breast at pre-op (P1) and y=[y₁, y₂, . . . , y₁₃₂₀]^(T) be theSPHARM coefficients for the post-op (P2) breast. Solving for linearleast squares optimization, a set of weights that determine thecontribution of each coefficient to the overall breast shape areobtained. A and B are diagonal matrices with x and y as their respectivediagonals. W is solved for using least squares optimization to determinethe transformation vector:

β=[β₁β₂. . . β₁₃₁₉ β₁₃₂₀]^(T).

AW=B

Where,

${A = \begin{bmatrix}x_{1} & \ldots & 0 \\ \vdots & \ddots & \vdots \\0 & \ldots & x_{1320}\end{bmatrix}},{B = \begin{bmatrix}y_{1} & \ldots & 0 \\ \vdots & \ddots & \vdots \\0 & \ldots & y_{1320}\end{bmatrix}},{W = \begin{bmatrix}\beta_{1} & \ldots & 0 \\ \vdots & \ddots & \vdots \\0 & \ldots & \beta_{1320}\end{bmatrix}}$

Least squares optimization fits the given linear model to find theweight vector β such that error is minimized.

min₆₂∥AW−B∥²

The estimated weight vector β is the transformation that when applied tox, results in y. Thus, when the transformation vector β is known, it canbe applied to the SPHARM coefficients of the pre-op breasts to obtainits post-op shape coefficients. To validate this hypothesis, SPHARMmodels for pre-op (P1) and their corresponding post-op (P2) breast imagepairs of individual patients who have undergone different cosmeticsurgical procedures (reduction, augmentation and mastopexy) aregenerated (see FIGS. 57A-C). The weight vector β obtained from leastsquares optimization to the P1 breast coefficients is applied. Thisresults in transformed P1 which is an estimate of the post-op breastshape P2, as demonstrated by the nearly zero RMSD values (see FIG. 56).

The squared spherical harmonic basis functions integrate to 4π insteadof 1, so a correction is added. The difference between the root meansquare error (RMSE) and the RMSD is that the RMSE is computed in thespatial domain while the RMSD is computed in the frequency domain. Theirvalues are similar to each other if comparing the same two objects.While the RMSD is relatively simple to compute from the coefficientsthemselves, other error measures (e.g., the mean absolute distance andother distance measures) that depend on the points should use a uniformsampling of the spherical parameterization, such as the iterativeicosahedron subdivision.

The Hausdorff distance, d_(H)(A, B), is the maximum distance of pointsin Set A to the nearest point in Set B and points in Set B to thenearest point in Set A. It is formally defined as:

${{d_{H}\left( {A,B} \right)} = {\max\left\{ {{\sup\limits_{a \in A}\inf\limits_{b \in B}{d\left( {a,b} \right)}},{\sup\limits_{b \in B}\inf\limits_{a \in A}{d\left( {a,b} \right)}}} \right\}}},$

where a and b are points of sets A and B, respectively, d(a, b) is theEuclidean distance between points a and b, sup is the supremum, and infis the infimum. This measure is used to evaluate if there is any pointin one object that is distant from the points of another object and viceversa.

The SPHARM coefficients contain both size and shape information, hencethe computed mean squared distance reflects differences in bothparameters. While the object can be normalized to remove the effect ofscale, in the case of breasts, size may be of interest since gravityplays a role in breast shape, especially for large breast sizes.

Referring to FIG. 47, a scatterplot of the first three principalcomponents for TRAM flap and implant reconstructed breasts is shown.Principal component analysis is a well-established dimensionalityreduction technique that transforms a set of observations with d numberof variables such that the first principal component contains thelargest possible variance, and each succeeding principal componentcontains the next largest variance until all the variance is accountedfor. The principal components are orthogonal to one another, and thetotal number of unique principal components may be less than the numberof variables. To perform principal component analysis, the mean of eachvariable is subtracted to center the data. Then the d ×d covariancematrix Σ is computed from the centered data. The eigenvalues andeigenvectors are computed from the covariance matrix, and theeigenvalues are sorted in a descending order along with theircorresponding eigenvectors. The centered data is transformed using thesorted eigenvectors. The first k principal components that explains 95%of the variance can be used for classification tasks.

Classification was performed to evaluate whether the SPHARM coefficientscan differentiate breasts that have undergone different reconstructionprocedures, such as TRAM flap and implant reconstructions. Threeclassifiers were used: k-nearest nearest algorithm, quadraticdiscriminant analysis, and Naïve Bayes. They are described as follows.

The k-nearest neighbor (k-NN) algorithm is a simple nonparametric methodfor assigning a class label to an object based on the class labels ofits k closest neighbors. Unlike decision trees and linear discriminants,k-NN does not require the explicit construction of a feature space.Theoretically, as the sample size tends to infinity, the error rate ofk-NN, under very mild conditions, tends to the Bayes optimal. Thesetting k is a user-defined constant that determines how a test point isclassified based on the most frequent label among the k nearest trainingpoints using the Euclidean distance. Essentially, the sample's predictedlabel R_(l) is C_(i) if the majority of the k nearest neighbors belongto C_(i),

S _(i) =C _(i) if (C _(i) /k>C _(j) /k).

Quadratic discriminant analysis uses a quadratic decision surface toseparate k classes. QDA is much like linear discriminant analysis,except that it assumes that the covariance matrix, Σk, is not identical,so the quadratic terms cannot be removed. The variables X are assumed tobe normally distributed for each class. The quadratic discriminantfunction is:

${{\delta_{k}(x)} = {{{- \frac{1}{2}}\log{❘{\sum k}❘}} - {\frac{1}{2}\left( {x - \mu_{k}} \right)^{T}{\sum{k^{- 1}\left( {x - \mu_{k}} \right)}}} + {\log\pi_{k}}}},$

where δ_(k)(x) is the estimated discriminant that the observation willbe in the kth class within the response variable given the predictorvariables x, Σk is the covariance matrix, and π_(k) is the priorprobability that an observation belongs to the kth class. Theobservation is assigned to the kth class depending on the largestdiscriminant score.

Naïve Bayes classifiers assume that variables (x=(x₁, . . . , x_(n)))are independent of one another. Each variable x_(i) contributesindependently to the probability that an observation belongs to the kthclass regardless of any correlations between different variables. Theprobability that an observation belongs to a class is given by

${p\left( C_{k} \middle| x \right)} = {\frac{{p\left( C_{k} \right)}{p\left( {x❘C_{k}} \right)}}{p(x)}.}$

Using the naïve independence assumption that

p(x _(i) |C _(k) x _(l) , . . . , x _(i)+1. . . , x _(n))=p(x_(i) |C_(k)),

for all θ, then the equation simplifies to:

${p\left( C_{k} \middle| x \right)} = {\frac{{p\left( C_{k} \right)}{\prod\limits_{i = 1}^{n}{p\left( {x_{i}❘C_{k}} \right)}}}{p(x)}.}$

Since (x) is constant given the input, the following classification rulemay be used:

${\hat{C}}_{k} = {\arg\max\limits_{C_{k}}{p\left( C_{k} \right)}{\prod\limits_{i = 1}^{n}{{p\left( {x_{i}❘C_{k}} \right)}.}}}$

An extension of Naïve Bayes for real-valued attributes is the GaussianNaïve Bayes, which assumes that the variables of each class are normallydistributed. The likelihood of the variables assumes a Gaussiandistribution:

${p\left( {x_{i}❘C_{k}} \right)} = {\frac{1}{\sqrt{2\pi\sigma_{y}^{2}}}\exp{\left( {- \frac{\left( {x_{i} - \mu_{C_{k}}} \right)^{2}}{2\sigma_{C_{k}}^{2}}} \right).}}$

The SPHARM models were tested to determine their applicability forpredicting breast shape after reconstruction based on exemplar data.This way if a new breast cancer patient comes in for a consultation withher surgeon to undergo reconstruction, the surgeon can show her apossible reconstruction outcome that is personalized (that is, she canbe shown her predicted reconstructed breast using her own pre-operativeimage), which may help in the decision making process. A diagram of theexample-based prediction method 1400 is shown in FIG. 14. The procedureis as follows: First, at step 1402, input the SPHARM breast model. Next,at step 1404, find the five closest preoperative breast models in thedatabase based on the RMSD value and create an average preoperativebreast model. Next, at step 1406, obtain the five correspondingpostoperative breast models and create an average postoperative breastmodel. Next, at step 1408, calculate the coefficient differences betweenthe average preoperative breast model and the average postoperativebreast model. Next, at step 1410, add the coefficient differences to theinput SPHARM breast model to simulate the predicted breast shape.

The breast modeling method was tested on a half sphere (200×200×100)with an open back. As shown in FIG. 15, the half sphere successfullyconverted to a SPHARM model using the method described in above. Thealgorithm closed the hole on the backside of the half sphere andparameterized the closed half sphere, and the SPHARM coefficients wascalculated using SPHARM-MAT. The root-mean-squared error between theoriginal half sphere points and the reconstructed half sphere pointsfrom the SPHARM coefficients was 1.023.

Next, the breast modeling method as described in the sections above wastested on real breast data. The results are shown in FIGS. 16A-C. Thereal texture or an artificial texture can be applied to the breastmodel. FIGS. 16A-C are SPHARM models of a small, a medium, and a largebreast.

The evaluation metrics were validated on synthetic half ellipticalmodels. The parameters of the synthetic models were based on a number ofSPHARM models that were generated from a dataset consisting of a numberof preoperative images of patients. The height, width, and depth of theSPHARM models and their proportionality to one another as well as thenumber of vertices and faces are summarized in FIG. 18A. The generatedSPHARM models are shown in FIG. 17. FIG. 17 is a table of syntheticmodels generated using different parameters. The RMSD, RMSE, andHausdorff distance between the models are shown in FIG. 18B, FIG. 19A,and FIG. 19B, respectively. For each increment of 20 mm in height,width, and depth between the models, the RMSD and RMSE increased byapproximately 10 mm, which is half of the increment value since most ofthe changes occurred only on the front side of the half sphere. The backside remained in place. The Hausdorff distance matched the incrementvalue.

The SPHARM coefficients were evaluated to determine their relation tospecific breast measurements that surgeons are familiar with, such asbreast height, width, depth (projection), and ptosis. Instead ofmodifying the coefficients directly, different synthetic modelsrepresenting different shapes and sizes were generated and their SPHARMcoefficients were compared. A half sphere and modified a singleparameter to simulate different heights, widths, depths, and ptosis(FIG. 20A and FIG. 20B) may be used. FIG. 20A is a table showingdifferent height, width, and projection parameter settings. FIG. 20B isa table showing different ptosis settings as simulated using Chen'smodel. Ptosis is one of the five global deformations implemented in themodel. FIGS. 21A-D are bar plots of the coefficient differences forheight, width, depth, and ptosis. After calculating the SPHARMcoefficient differences between two synthetic models with a heightparameter of 100 versus 120, two coefficients, c_(z1) ⁻¹ and c_(z1) ¹,exhibited 91.25% of the total coefficient value differences. These twocoefficients had equal value differences as a symmetric object was used.For two models with a width parameter of 100 versus 120, coefficientsc_(y1) ⁻¹ and c_(y1) ¹ contained 92.55% of the total coefficient valuedifferences. These two coefficients were also equal due to the symmetricobject. For a depth parameter of 50 versus 100, coefficient c_(x1) ⁰accounted for 54.71% of the total coefficient differences andcoefficient c_(x2) ⁰ accounted for 26.87%. For a ptosis parameter of 0versus 1, coefficient c_(z1) ⁰ accounted for 54.71% of the coefficientdifferences and coefficient c_(z2) ⁰ accounted for 26.87% of thedifference (FIGS. 21A-D). The following coefficients after the first twolargest coefficient differences (for height, width, depth, and ptosis)each accounted for less than 6.3% of the total coefficient differences.In various embodiments, breast shape changes may be related to a fewSPHARM coefficients. Therefore, at least two coefficients may be changedto modify the size or shape of a model. FIGS. 22A-D shows how thecoefficients are linearly related to changes in the parameters.

After finding the SPHARM coefficients most associated with height,depth, projection, and ptosis, the Pearson correlation between realbreast measurements and the selected SPHARM coefficients was evaluated.A unit change in the coefficients may modify the breast shape. Thevolume of the modeled breast against the actual breast volume asmeasured using Passalis's method, which employs a Coons patch torepresent the back wall of the breast was compared.

For example, using a dataset containing 87 preoperative images ofpatients, 161 of the 174 breasts were successfully converted to SPHARMmodels. The relationship between the ground truth ptosis rating providedby a surgeon versus the ptosis coefficients, the measured height versusthe height coefficient, the measured width versus the width coefficient,the measured projection versus depth coefficient, and the volumemeasured in customized software versus the SPHARM model volume for the161 SPHARM models was evaluated. Projection is defined as the distancefrom the most projected point on the breast to its corresponding pointon the back side of the breast. The volume computed in customizedsoftware is the space between the breast surface and the estimated Coonspatch. The volume for the SPHARM models is the space contained withinthe closed mesh object that was computed in MATLAB.

Referring to FIGS. 24A-B, tables of the P-values indicating whether thecolumn means are significantly different are shown. Separating theptosis coefficients based on the ptosis rating, the ptosis coefficientsof ptosis grade 0 was significantly different from all the other ptosisgrades. The first ptosis coefficient (c_(z1) ⁰ of ptosis grade 2 was notsignificantly different from ptosis grades 1 and 3. The second ptosiscoefficient (c_(z2) ⁰ of ptosis grade 2 was not significantly differentfrom ptosis grade 1, but was significantly different from ptosis grade 3(FIGS. 24A-B). Ptosis grade 1 for both ptosis coefficients wassignificantly different from those of ptosis grade 3. The ptosiscoefficients' average value of each grade may be progressively lowerthan that of the previous grade (FIG. 23). With further work, the ptosiscoefficients could potentially be used to objectively grade ptosis andto provide consistent results.

Referring to FIGS. 25A-D, scatterplots of the height coefficient versusheight, the width coefficient versus width, the depth coefficient versusprojection, and the SPHARM volume versus the volume calculated incustomized software are shown. The model height was strongly correlatedwith the height coefficient (R²=0.9). The width coefficient had a lower,but still strong, correlation with the model width (R²=0.7). The breastnaturally curves toward the back on the lateral side of the breast, butthe model width measurement does not account for this curvature, whichmay explain the lower correlation in width. The correlation between theSPHARM depth coefficient and the measured projection was moderatelypositive with R²=0.4. The projection measurement versus its coefficienthad a much lower correlation, since it is both highly dependent on thelocation of the highest projected point and the curvature of thebackside of the breast, which are both variable. The SPHARM volume wasstrongly correlated with the customized volume (R²=0.9), and the ratiowas almost 1:1. All correlations were significant based on the Student'st distribution (p<0.001).

The SPHARM degree is a user-defined constant that determines the numberof coefficients that is used to represent the breast model. While ahigher degree increases the number of coefficients and leads to a moredetailed reconstruction, there is also the possibility of overfittingwith increasing degrees. A degree to use based on a dataset of 87patients, which is described above. For example, of the 174 breasts, 161breasts (92.5%) were successfully converted to SPHARM models using themanually selected fiducial points, which may be assigned as the groundtruth dataset. Thirteen of the breasts did not convert to SPHARM modelsdue to non-manifold vertices in the mesh. The ground truth breastobjects consisted of, on average, 10654±3536 vertices and 21305±7073faces. The smallest breast object had 4113 vertices and 8222 faces,which sets a limitation on the highest possible degree to 63 degrees orlower ((63+1)=4096 vertices) for calculating the SPHARM coefficients.The triangles can be subdivided to increase the number of vertices andthus increase the maximum degree and reduce overfitting results. Toobtain the best degree to represent the breast data, the SPHARMcoefficients for degrees 10 to 50 at intervals of 10 for each inputbreast mesh (consisting of vertices and faces) in the dataset werecalculated. The vertices of the reconstructed SPHARM model were comparedto the vertices of the original breast mesh using the method describedabove. Since the coefficients are well-fitted to the input breast mesh,the reconstructed SPHARM models were given new spherical coordinates toestimate the locations of the face centroids in the input breast mesh.The rationale behind generating a model with new vertices is to evaluateif the coefficients generated with the degree used are accurate enoughto generate a model close to the original without overfitting. Theroot-mean-square error (RMSE) and the Hausdorff distance between thevertices of the reconstructed SPHARM model and the face centroids of theoriginal data were measured to determine the degree that resulted in themost accurate reconstructed SPHARM model.

As shown in FIG. 26A, it was found for the data, that degrees 10-30 gavesatisfactory results based on the RMSE. All breast samples had RMSEvalues less than 10 mm when using degrees 10 to 30. Using a degree of40, 9.9% of the breast samples showed signs of overfitting, while degree50 led to overfitting in 50.9% of the breast samples, as exemplified inFIG. 26B. However, the Hausdorff distance showed that some breastsamples had a few points that were out of place for degree 30 (FIG.26B). FIGS. 27A-C are depictions of example SPHARM models based ondegree 1, 20, and 50 using level 4 icosahedral subdivision, inaccordance with aspects of the present disclosure.

The quality of the results of the different degrees against the SPHARMcoefficients that were identified to be most associated with height(c_(z1) ⁻¹), width (c_(y1) ¹) projection/depth (c_(x1) ⁰), and ptosis(c_(z1) ⁰ and c_(z2) ⁰) was also evaluated. The magnitude of the SPHARMcoefficients associated with height, width, depth, and ptosis wascalculated for different degrees for each breast sample and the averagevalues are shown in FIG. 28. Using the Shapiro Wilk's normality test,the height and width coefficients were found to have non-normaldistributions for all degrees (p<0.01). The depth and ptosiscoefficients were found to have normal distributions for degrees 10 to30 (p>0.05) but were not normally distributed for degrees 40 and 50(p<0.001). Therefore, the Wilcoxon rank-sum test may be used, anon-parametric test, for evaluating whether the coefficients haddifferent values between different degrees. The Wilcoxon rank-sum testshowed that the coefficients were not significantly different betweendegrees 10-40, but degrees 10-40 were all significantly different fromthe coefficients calculated for degree 50 as shown in FIG. 29. Since thecoefficients from degrees 10 to 30 are similar with <1% differencerelative to the coefficient value (FIG. 30), there is flexibility inselecting the degree to represent the model. Different objects may becompared based on these coefficients even if the degree used tocalculate the coefficients is different. The degree level of 20 for allsimulations was used. Not only does degree 20 take less than half thetime to compute than degree 30, it is also relatively accurately. TheHausdorff distance error also showed that degree 30 had a few caseswhere overfitting occurred.

In order to crop the breasts, two fiducial points had to be manuallyidentified: the transition point and the lateral point. A test wasconducted to evaluate the robustness of the algorithm to form a SPHARMmodel if the transition point and the lateral point were placed indifferent locations and to identify at what step in the algorithm itfails. The modeling algorithm can be divided into five major steps: (1)midline (ML) and inframammary fold or inferior breast-chest contour(IMF) detection, (2) breast cropping, (3) creating a closed mesh object,(4) spherical parameterization, and (5) SPHARM expansion.

The region in which the transition point and lateral point may belocated were identified based on certain criteria, and only the fourcorners of this region, which are called the extremities, were tested,as presented in FIGS. 31A-C. FIG. 31A shows the lateral point selectionarea for one breast. FIG. 31B shows the transition point selection areafor both breasts. FIG. 31C shows the lateral point selection for theopposite breast. The dots represent the ground truth selection. Thecriteria for the transition point region were as follows: 1) area belowthe sternal-notch, 2) above the midline point and above the mostprojected point, and 3) within the middle 50% of the breast width. Thecriteria for the lateral point were as follows: 1) area below thearmpits, 2) above the lowest point of the inframammary fold, and 3)within 37° (cos⁻¹ 0.8) of the horizontal direction.

The true transition point and the lateral point will always be withinthese defined regions, and a properly trained user will not select thefiducial point outside of these regions. The software itself can bedesigned to limit the choices to these regions. The four corners of theregion identified for the transition point and lateral point weretested. The four transition points (TP) and the four lateral points (LP)were paired as follows:

Set 0: Ground truth (Manually selected TP and LP)

Set 1: Top-medial TP and top-front LP

Set 2: Top-lateral TP and top-back LP

Set 3: Bottom-lateral TP and bottom-back LP

Set 4: Bottom-medial TP and bottom-front LP

Referring to FIG. 32, a patient's breasts based on these five pairs ofpoints may be modeled. FIG. 33 shows the rate of successful processingat each step of the algorithm for different landmark positions for themodeled breast. FIG. 34 shows the Euclidean distance of the userselection relative to the ground truth selection for ten (10) images.FIG. 35 is a table of the Euclidean distance of the corner (extreme)selections relative to the ground truth selection for 87 images. FIG. 36is a diagram of the lateral point (LP) location of the different setsrelative to the ground truth (set 1). FIG. 37 is a table of the rate ofsuccessful processing at each step of the algorithm for differentlateral point locations based on 161 breasts. FIG. 38 is a diagram ofthe transition point (TP) location of the different sets relative to theground truth (Set 1). FIG. 39 is a table of the rate of successfulprocessing at each step of the algorithm for different transition pointlocations based on 161 breasts. FIG. 40 is a table of the statistics onthe projection percent difference between the ground truth selection andshifting the TP three (3) and six (6) millimeters in differentdirections including lateral, medial, down and up. FIG. 41 is a tabledepicting statistics on the volume percent difference between the groundtruth selection and shifting the TP three (3) and six (6) millimeters indifferent directions including lateral, medial, down and up. FIG. 42 isa table depicting statistics on the projection percent differencebetween the ground truth selection and shifting the LP three (3) and six(6) millimeters in different directions including down, up, forward andbackward. FIG. 43 is a table depicting statistics on the volume percentdifference between the ground truth selection and shifting the LP three(3) and six (6) millimeters in different directions including down, up,forward and backward

Referring to FIGS. 44A-B, a program may be used to adjust the height,width, depth, and ptosis to any loaded breast data. For example, aMATLAB application was created that allows any user to easily applydifferent settings for adjusting height, width, projection, and ptosisto any loaded breast data that has SPHARM coefficients computed for it.The original texture or a generic texture can be mapped to the breastmodel. The application takes less than half a second to process theinput parameters and display the modified breast model.

Referring to FIGS. 45A-B, two commonly performed types of breastreconstruction TRAM flap and implant reconstructions are shown. TRAMflaps tend to give breasts a smooth teardrop shape that looks morenatural (FIG. 45A), while implants create round and protruded shapedbreasts (FIG. 45B).

Referring to FIGS. 46A-B, comparisons using sets of confusion matricesfor k-nearest neighbor (k-NN) classification (k=3 and 5), quadraticdiscriminate analysis (QDA), and Naïve Bayes classifier using the SPHARMcoefficients versus matrices for k-NN classification, quadraticdiscriminate analysis, Naïve Bayes classifier using BMI, breast volume,and breast dimensions were made. Classification was performed to see ifthe reconstructed breasts can be differentiated by reconstruction typeusing the SPHARM coefficients as feature vectors. The SPHARMcoefficients can differentiate the breasts based on their shapes, whichcan later be applied in classifying natural breast shapes.

For a standard of comparison, classification using BMI was performed,breast height, width, projection, and volume as measured usingcustomized software. The results shown in FIGS. 46A-B indicated that theSPHARM coefficients has more discriminative ability than only using theBMI, breast height, width, depth, and volume.

Referring to FIG. 48, a set of images of true versus predicted classreconstructed breasts according to a TRAM flap or implant are shown.

A template breast model can be created from a set of breast models thatare similar in shape using the RMSD. Two examples of the average breastobject are shown below. One TRAM flap reconstructed breast model wasselected, and four other breast models that were similar in shape (outof 28 TRAM reconstructed breasts) based on the coefficients were alsoselected. The five breast models shown in FIGS. 49A-B were averagedtogether to form an average breast model shown in FIG. 51A. The originalimages are shown in FIG. 49A and the SPHARM models with the originaltexture applied are shown in FIG. 49B. All right breasts were mirroredin order to compare with the left breasts. The RMSDs were between 5.51and 7.16 for the first TRAM flap reconstructed breast versus the otherfour breasts. Relative to the average breast shape, the RMSDs of thefive breasts were between 3.83 and 4.66. In another example, an implantreconstructed breast model and its four nearest implant reconstructedbreast models (out of 23 implant breasts) were averaged together (FIGS.50A-B) to form an average breast model (FIG. 51B). FIG. 50A shows theoriginal images and FIG. 50B shows the SPHARM models with the originaltexture applied. The RMSDs were between 3.89 and 5.54 for the implantreconstructed breast versus the other four implant reconstructedbreasts. Relative to the average breast shape, the RMSDs of the fivebreasts were between 2.57 and 4.38. After demonstrating with twoexamples that an average breast representing a set of similar breastshapes can be created, the following section shows how the averagebreast may be used for predictive modeling.

Referring to FIGS. 52A-D, tables of data for a set of patients are showndescribing the types of operations undergone (FIG. 52A), the number ofpatients, the RMSD of various preoperative and postoperative breastcomparisons (FIG. 52B), the RMSD for various BMI (FIG. 52C), and the HDfor various BMI (FIG. 52D). FIG. 52A is a table of the proceduresconducted on the left and right breast of each patient and the averageRMSD for each set of patients with the same procedures. FIG. 52B is atable of the statistics on the RMSD comparing preoperative,postoperative, and predicted breast shapes. FIG. 52B is a table of thestatistics on the RMSD comparing preoperative, postoperative andpredicted breast shapes. FIG. 52C is a table of the RMSD between thetrue postoperative models and the predicted models organized into threeBMI groups. FIG. 52D is a table of the HD between the true postoperativemodels and the predicted models organized into three BMI groups.

Referring to FIG. 53, postoperative models may be predicted. In variousembodiments, by using the average breast model described in the previoussection, individualistic features may be averaged and/or smoothed out,such as the nipple. In this way, a depression does not get added in thepredicted breast, as shown in row 3 of FIG. 53. The RMSD between thepredicted and the actual postoperative model also improved. Only thefirst 27 coefficients were added in order to avoid the depression butfound that the RMSD was lower when using the average object approach, aspresented in FIG. 52B and exemplified in FIG. 53. For example, aprediction had a RMSD of 6.17 between the true postoperative model andthe predicted model as shown in FIG. 54. The five closest breast modelshad RMSDs between 6.83 and 9.35, so there were relatively similar shapedbreasts in the database. On the other hand, in a prediction, thereference image's closest breast models in the database had RMSDsbetween 14.99 and 19.04, so there just wasn't a similar breast shapeavailable in the database to make a more accurate prediction (FIG. 55).When the prediction models were separated by BMI, the RMSDs between thetrue postoperative model and the predicted model were lower for BMI lessthan 25 and higher for BMI greater than 30 (FIG. 52C). The averageHausdorff distance was also larger for patients with BMI greater than 30than the patients with lower BMI (FIG. 52D). The Hausdorff distance wascomputed between the true postoperative SPHARM model and the predictedmodel that were sampled using a level 5 icosahedral subdivision (10242vertices and 20480 faces). Larger breasts have poorer predictions sincethere can be more variability in size and shape than for small breasts.In various embodiments, SPHARM models may be used to predict outcomes,which may be improved with a larger dataset and matching by age, BMI,breast volume, reconstruction type, smoking history, and other factorsincluding patient preferences. All of these factors are discussed duringpatient consultations, which can be used to help predict the model.

In various embodiments, a method to model the breast that can be used toanalyze, compare, and modify its shape, is shown. The algorithm may berobust to small differences in the point selection of the transitionpoint and lateral point. Results on classification for different breastreconstruction types are shown, creating average breast objects that canrepresent a particular shape, and predictive modeling. Thethree-dimensional model based on SPHARM and its further development willprovide a state-of-the-art surgical planning tool for surgeons tovisualize and interactively evaluate the morphology of the breast. Itwill also help patients in making more informed decisions. In addition,for a patient who is yet to undergo breast reconstruction, her breastscan be shape matched to the preoperative breasts of previousreconstruction patients and then shown their post-surgical outcomes,which may help the patient mentally prepare for possible outcomes.

In various embodiments, a standard may be developed for automaticallydetecting the lateral and transition points to maintain consistency(increase precision), if not accuracy, across different time points inthe reconstructive process (as the patient is imaged every three monthsbefore and after mastectomy and reconstruction) as well as acrossdifferent patients. The model may also be reconnected with the torso toevaluate the overall appearance of the breast in relation to the humanbody. As demonstrated in the classification results, the SPHARMcoefficients has potential for classifying different breast shapes.There are several natural breast shapes that have been identified forwomen. They may serve as a starting point for helping to objectivelycategorize the shape of a woman's breasts in order to select the rightbra size and type that would fit comfortably.

In various embodiments, a method to predict surgical outcome may includeacquiring a 3D pre-op image of a patient, looking at database forsimilar demographics (e.g., age etc.) and breast size and shape, wherethe database also includes pre-op and post-op 3D images, find postoperation image of those patients, determine new SPHARM coefficients,applying the SPHARM coefficients to the 3D pre-op image of the patient,and morphing the breast based on the new SPHARM coefficients.

In various embodiments, a method to predict surgical outcome may includethe generation of template breast shapes that can be then used topredict and/or visualize the breast shape for women seeking a particularoption, or to compare different options.

Referring to FIGS. 57A-D, images of frontal and lateral views of pre-op(P1), post-op (P2), estimate (E) and overlay of estimate on the post-opbreast for four breasts in the input test set are shown. The post-opimages in FIGS. 57A-D are actual reconstruction results obtained byimaging the patient after the surgery at 18 months in a consultationtimeline.

In various embodiments, deep learning/AI/machine learning algorithms maybe used in the above method to predict surgical outcomes. The predictingmay include using a machine learning algorithm, where training datainputs include for example, pre and post operation image data and/orpatient demographic data. Machine learning algorithms may include, forexample, a neural network, random forest regression, linear regression(LR), ridge regression (RR), least-angle regression (LARS), and/or leastabsolute shrinkage and selection operator regression (LASSO). Themachine learning algorithms may be executed on the controller (see FIG.3B), and/or on a remote computing system.

In the clinical setting, a 3D surface image of pre-op breast isavailable before the surgery is scheduled and the surgical option underconsideration is known, or to be determined. The shape change of breastspre- and post-surgery is dependent on surgery type and other medicalparameters such as ptosis grade, implant size and weight, skinelasticity. It is not feasible to compute a single generalizedtransformation for pre-op breast shape to the expected post-op shape forall surgery types. A data-driven approach may be employed to estimatethe transformation vector using nonlinear regression for any changes inbreast shape, including natural (e.g. aging, pregnancy, or otherdeformities), or surgical.”

SPHARM coefficients of the pre-op breast and the transformation vectorrequired to obtain its post-op shape using least squares optimizationfor twenty-one pre-op and its corresponding post op breasts and traineda random forest regression function to learn the non-linear relationshipbetween the transformation vectors, are computed. Random regressionforest is an ensemble learning method that is a popular model fornon-linear regression. Random forest regression efficiently preformsregression for multivariate data. For example, the regression functionwas trained using bootstrap sample of 21 breasts, with 1320 SPHARMcoefficients, x=[x₁, x₂, . . . x₁₃₂₀]^(T) of the pre-op breast as theinput features and the transformation vector β=[β₁ β₂. . . β₁₃₁₉β₁₃₂₀]^(T) as the regression output. Random regression forest is made ofseveral individual regression trees. A regression tree is recursivelyconstructed such that at each node the training data is split on arandomly chosen feature variable so that entropy at the node isminimized. In a regression tree the entropy of the feature densitiesassociated with different nodes decreases when going from the roottowards the leaves. When presented with an unseen test data, the randomforest simply averages the results from individual regression trees topredict the output. The transformation obtained from the regression tothe pre-op (P1) breast coefficients to obtain the estimation of post-opshape (see FIG. 57D) was applied.

In various embodiments, the method further includes identifyingdifferent types of natural breast shapes and contours, shapes related tobreast diseases, and outcomes of surgical procedures, includingreconstructed breasts (at least one of autologous or implantreconstructed breasts), and cosmetic procedures (augmentation,reduction, mastopexy), based on spherical harmonic coefficients.

In various embodiments, a method of predicting includes creating ageneral shape template from images of several women. In an embodiment,the general template may be created using images of other breasts from agroup of women (similar in demographics such as BMI, age, etc.). In anembodiment, the general template may be created for specific breastconditions using data from large groups of women (i.e. not the few thathave the most similar shape).

It is contemplated that the database for 3D images of patient withsimilar demographics are not limited to age, breast size, or breastshape, and may contain other demographic data.

FIG. 58 is a block diagram of a method for modeling a breast, inaccordance with the present disclosure.

The flow diagram of FIG. 58 shows a computer implemented method 900 formodeling a breast. Persons skilled in the art will appreciate that oneor more operations of the method 5800 may be performed in a differentorder, repeated, and/or omitted without departing from the scope of thedisclosure. In some methods in accordance with this disclosure, some orall of the operations in the illustrated method 5800 can be operated onthe controller 200 (see FIG. 3B). Other variations are contemplated tobe within the scope of the disclosure. The operations of FIG. 58 will bedescribed with respect to a computing device, e.g., controller 200 ofsystem 300 (FIG. 3A), or any other suitable computing system device orlocation thereof including a remotely disposed computing device. It willbe understood that the illustrated operations are applicable to othersystems and components thereof as well.

Initially, at step 5802, the method receives a 3D image (e.g., apre-operative image), which includes a breast. Next, at step 5804, themethod identifies the breast in the 3D image. Next, at step 5806, themethod extracts 3D image data of the breast from the 3D image. Next, atstep 5808, the method forms a closed object using 3D image data of thebreast to create a zero-genus surface. Forming of a closed object mayinclude identifying holes in a mesh by finding boundary edges, which areedges that are not shared by two faces, calculating the angle betweenadjacent boundary edges at a vertex, locating the smallest angle andcreating a new triangle at the vertex, wherein a location of newvertices is determined by an average edge length and the shortestdirection to close a gap across two meshes, computing a distance betweenevery newly created vertex and every related boundary vertex, in a casewhere the distance between them is less than a predetermined threshold,they are merged, and updating the mesh based on the computed distance.Next, at step 5810, the method maps the 3D image data of the breast to apredefined template using spherical co-ordinates (e.g., sphericalparameterization). Next, at step 5812, the method determines a 3Dspherical harmonic descriptor of the 3D image data of the breast, forexample, based on minimization. The method may include identifyingparameters of the 3D spherical harmonic descriptor that representanatomical breast parameters including height, width, depth, and/orptosis. The method may include identifying different types of breastshapes, such as autologous and/or implant reconstructed breast or acombination of autologous and implant breasts, based on sphericalharmonic (SPHARM) coefficients.

The method may include predicting a post-operative breast shape from the3D image based on the 3D spherical harmonic (SPHARM) model andoutputting a predicted 3D image based on the predicted post-operativebreast shape. The method can also be used for predicting any naturalbreast shape change such as, for example, due to the aging process andweight loss/gain, or pathological breast shape change such asdeformities, or any surgical alterations.

The method may include searching a database, of pre-operative andpost-operative 3D images, for a 3D image of at least one second patientwith similar demographics or medical history, to the received thepatient of the 3D image, determining SPHARM coefficients of the received3D image, locating a pre-operative 3D image of a second patient with asimilar age, breast size, and/or breast shape based on the SPHARMcoefficients. The method may further include locating a post-operative3D image of the second patient, generating an average pre-operative 3Dimage based on the pre-operative 3D images, generating an averagepost-operative 3D image based on the post-operative 3D images,determining SPHARM coefficients of at least one of an averagepre-operative 3D image or a located post-operative 3D image, determiningSPHARM coefficients of at least one of an average post-operative 3Dimage or a located post-operative 3D image. The method may furtherinclude determining a difference between SPHARM coefficients of thereceived 3D image or the average pre-operative image and SPHARMcoefficients of the average post-operative 3D image, applying thedifference in SPHARM coefficients to the received 3D image, and morphingthe breast of the received 3D image based on the determined SPHARMcoefficients.

The embodiments disclosed herein are examples of the disclosure and maybe embodied in various forms. For instance, although certain embodimentsherein are described as separate embodiments, each of the embodimentsherein may be combined with one or more of the other embodiments herein.Specific structural and functional details disclosed herein are not tobe interpreted as limiting, but as a basis for the claims and as arepresentative basis for teaching one skilled in the art to variouslyemploy the present disclosure in virtually any appropriately detailedstructure. Like reference numerals may refer to similar or identicalelements throughout the description of the drawings.

The phrases “in an embodiment,” “in embodiments,” “in variousembodiments,” “in some embodiments,” or “in other embodiments” may eachrefer to one or more of the same or different embodiments in accordancewith the present disclosure. A phrase in the form “A or B” means “(A),(B), or (A and B).” A phrase in the form “at least one of A, B, or C”means “(A); (B); (C); (A and B); (A and C); (B and C); or (A, B, andC).”

Any of the herein described methods, programs, algorithms or codes maybe converted to, or expressed in, a programming language or computerprogram. The terms “programming language” and “computer program,” asused herein, each include any language used to specify instructions to acomputer, and include (but is not limited to) the following languagesand their derivatives: Assembler, Basic, Batch files, BCPL, C, C+, C++,Delphi, Fortran, Java, JavaScript, machine code, operating systemcommand languages, Pascal, Perl, PL1, scripting languages, Visual Basic,metalanguages which themselves specify programs, and all first, second,third, fourth, fifth, or further generation computer languages. Alsoincluded are database and other data schemas, and any othermeta-languages. No distinction is made between languages which areinterpreted, compiled, or use both compiled and interpreted approaches.No distinction is made between compiled and source versions of aprogram. Thus, reference to a program, where the programming languagecould exist in more than one state (such as source, compiled, object, orlinked) is a reference to any and all such states. Reference to aprogram may encompass the actual instructions and/or the intent of thoseinstructions.

It should be understood that the foregoing description is onlyillustrative of the present disclosure. Various alternatives andmodifications can be devised by those skilled in the art withoutdeparting from the disclosure. Accordingly, the present disclosure isintended to embrace all such alternatives, modifications and variances.The embodiments described with reference to the attached drawingdrawings are presented only to demonstrate certain examples of thedisclosure. Other elements, steps, methods, and techniques that areinsubstantially different from those described above and/or in theappended claims are also intended to be within the scope of thedisclosure.

What is claimed is:
 1. A computer implemented method of modeling abreast shape, the method comprising: receiving a 3D image including abreast; identifying the breast in the 3D image; extracting 3D image dataof the breast from the 3D image; forming a closed object using the 3Dimage data of the breast to create a zero-genus surface; mapping the 3Dimage data of the breast to a predefined template using sphericalcoordinates; and determining a 3D spherical harmonic descriptor of the3D image data of the breast.
 2. The method of claim 1, wherein themethod further includes identifying parameters of the 3D sphericalharmonic descriptor that represent anatomical breast parametersincluding at least one of a height, a width, a depth, or ptosis.
 3. Themethod of claim 1, wherein the method further includes identifyingdifferent types of breast shapes, including at least one of a naturalbreast, a surgically altered breast, an autologous breast, or an implantreconstructed breast, or a combination of autologous and implantbreasts, based on spherical harmonic (SPHARM) coefficients.
 4. Themethod of claim 1, wherein the 3D image is a patient's preoperativeimage; and the method further includes: predicting a post-operativebreast shape from the 3D image based on the 3D SPHARM model; andoutputting a predicted 3D image based on the predicted post-operativebreast shape.
 5. The method of claim 4, wherein the predicting includes:searching a database for a 3D image of at least one second patient withsimilar demographics or medical history, to the received the patient ofthe 3D image, wherein the database includes pre-operative andpost-operative 3D images; determining SPHARM coefficients of thereceived 3D image; locating a pre-operative 3D image of at least onesecond patient with at least one of a similar age, breast size, orbreast shape based on the SPHARM coefficients; locating a post-operative3D image of the at least one second patient; generating an averagepre-operative 3D image based on the pre-operative 3D images; generatingan average post-operative 3D image based on the post-operative 3Dimages; determining SPHARM coefficients of at least one of an averagepre-operative 3D image or a located post-operative 3D image; determiningSPHARM coefficients of at least one of an average post-operative 3Dimage or a located post-operative 3D image; determining a differencebetween SPHARM coefficients of the received 3D image or the averagepre-operative image and SPHARM coefficients of the averagepost-operative 3D image; applying the difference in SPHARM coefficientsto the received 3D image; and morphing the breast of the received 3Dimage based on the determined SPHARM coefficients.
 6. The method ofclaim 4, wherein the predicting includes: identifying, in a database, apost-op 3D image of at least one second patient with similardemographics or medical history, or breast shape to the patient of thereceived 3D image, wherein the database includes post-operative 3Dimages of breasts; generating a template post-operative 3D image basedon the identified post-operative images to represent a particularoutcome; determining SPHARM coefficients of the received 3D image andSPHARM coefficients of the template; determining a difference betweenthe SPHARM coefficients of the received 3D image and the SPHARMcoefficients of the template; applying the difference in SPHARMcoefficients to the received 3D image; and morphing the breast of thereceived 3D image based on the determined SPHARM coefficients.
 7. Themethod of claim 4, wherein the predicting includes using a machinelearning algorithm, where training data inputs include at least one ofpre operation image data, pre operation model data, post operation imagedata, post operation model data, or patient demographic data.
 8. Themethod of claim 7, wherein the machine learning algorithm includes atleast one of a neural network, random forest regression, linearregression (LR), ridge regression (RR), least-angle regression (LARS),or least absolute shrinkage and selection operator regression (LASSO).9. The method of claim 4, wherein the method includes identifyingdifferent types of breast shapes based on position including at leastone of upright, supine prone, or any position there between, generatingposition specific templates, and wherein the outputting is based onpatient position including at least one of upright, supine, prone, orany position there between.
 10. The method of claim 9, wherein thedifferent types of breast shapes include at least one of natural,unnatural, surgically altered, or aged.
 11. The method of claim 1,wherein the forming of a closed object includes: identifying holes in afirst mesh by finding boundary edges, which are edges that are notshared by two faces; calculating the angle between adjacent boundaryedges at a vertex; locating the smallest angle and creating a newtriangle at the vertex; creating a second mesh to substantially fill theidentified holes, wherein a location of a second vertex is determined byan average edge length and a shortest direction to close a gap acrossthe two meshes; computing a distance between every newly created vertexand every related boundary vertex, in a case where the distance betweenthem is less than a predetermined threshold they are merged; andupdating the mesh based on the computed distance.
 12. The method ofclaim 1, wherein the method further includes identifying different typesof breast shapes, including at least one of natural breast shape,cosmetically altered breast shape, surgically reconstructed breastshape, reduction mammoplasty, reduction mastopexy, augmentationmammoplasty, augmentation mastopexy, or correction of any breast shapedeformities, based on spherical harmonic coefficients.
 13. A system formodeling a breast shape, the system comprising: a processor; and amemory, including instructions, which when executed by the processor,cause the system to: receive a 3D image including a breast; identify thebreast in the 3D image; extract 3D image data of the breast from the 3Dimage; form a closed object using the 3D image data of the breast tocreate a zero-genus surface; map the 3D image data of the breast to apredefined template using spherical coordinates; and determine a 3Dspherical harmonic descriptor of the 3D image data of the breast. 14.The system of claim 13, wherein the instructions, when executed, furthercause the system to identify parameters of the 3D spherical harmonicdescriptor that represent anatomical breast parameters including atleast one of a height, a width, a depth, or ptosis.
 15. The system ofclaim 13, wherein the instructions, when executed, further cause thesystem to identify different types of breast shapes, including at leastone of autologous or implant reconstructed breast or a combination ofautologous and implant breasts, based on spherical harmonic (SPHARM)coefficients.
 16. The system of claim 13, wherein the 3D image is apatient's preoperative image; and wherein the instructions, whenexecuted, further cause the system to: predict a post-operative breastshape from the 3D image based on the 3D SPHARM model; and output apredicted 3D image based on the predicted post-operative breast shape.17. The system of claim 16, wherein when predicting, the instructions,when executed, further cause the system to: search a database for a 3Dimage of at least one second patient with similar demographics ormedical history, to the received the patient of the 3D image, whereinthe database includes pre-operative and post-operative 3D images;determine SPHARM coefficients of the received 3D image; locate apre-operative 3D image of at least one second patient with at least oneof a similar age, breast size, or breast shape based on the SPHARMcoefficients; locate a post-operative 3D image of the at least onesecond patient; generate an average pre-operative 3D image based on thepre-operative 3D images; generate an average post-operative 3D imagebased on the post-operative 3D images; determine SPHARM coefficients ofat least one of the average pre-operative 3D image determine SPHARMcoefficients of at least one of the average post-operative 3D image or alocated post-operative 3D image; determine a difference between SPHARMcoefficients of the received 3D image or the average pre-operative imageand SPHARM coefficients of the average post-operative 3D image; applythe difference in SPHARM coefficients to the received 3D image; andmorph the breast of the received 3D image based on the determined SPHARMcoefficients.
 18. The system of claim 16, wherein when predicting, theinstructions, when executed, further cause the system to: identify, in adatabase, a post-op 3D image of at least one second patient with similardemographics or medical history, or breast shape to the patient of thereceived 3D image, wherein the database includes post-operative 3Dimages of breasts; generate a template post-operative 3D image based onthe identified post-operative images to represent a particular outcome;determine SPHARM coefficients of the received 3D image and the SPHARMcoefficients of the template; determine a difference between the SPHARMcoefficients of the received 3D image and the SPHARM coefficients of thetemplate; apply the difference in SPHARM coefficients to the received 3Dimage; and morph the breast of the received 3D image based on thedetermined SPHARM coefficients.
 19. The system of claim 16, wherein thepredicting includes using a machine learning algorithm, where trainingdata inputs include at least one of pre and post operation image data orpatient demographic data, wherein the machine learning algorithmincludes at least one of a neural network, random forest regression,linear regression (LR), ridge regression (RR), least-angle regression(LARS), or least absolute shrinkage and selection operator regression(LASSO).
 20. A non-transitory storage medium that stores a programcausing a computer to execute a method for modeling a breast shape, themethod comprising: receiving a 3D image including a breast; identifyingthe breast in the 3D image; extracting 3D image data of the breast fromthe 3D image; forming a closed object using the 3D image data of thebreast to create a zero-genus surface; mapping the 3D image data of thebreast to a predefined template using spherical coordinates; anddetermining a 3D spherical harmonic descriptor of the 3D image data ofthe breast.